QTM 385 - Experimental Methods

Lecture 22 - Survey Experiments for Sensitive Topics

Danilo Freire
danilo.freire@emory.edu

Emory University

Hello, everyone! 😉

Brief recap 📚

Survey experiments

Core components

  • Survey experiments combine random assignment with survey methods to study attitudes
  • Main applications: behavioural economics, psychology, marketing, political behaviour, and public opinion
  • Core design variations:
    • Presence/absence of stimuli
    • Dosage levels of treatment intensity
    • Qualitative variations in treatment content
  • Validation methods:
    • Manipulation checks post-treatment
    • Placebo treatments for specificity testing
    • Non-equivalent outcomes for effect containment
  • Common implementations:
    • Question wording manipulations
    • Vignette designs with randomised attributes
    • Audio or video stimuli

Today’s plan 📅

Survey experiments with a twist

  • What to do when subjects have an incentive to lie?
  • Sensitive topics are often difficult to study
  • Social desirability bias is a common problem in survey research
  • We need special techniques to study these problems
  • List experiments, randomised response technique, endorsement experiments, and conjoint experiments are possible solutions
  • Maintains plausible deniability for respondents
  • List experiments measure prevalence indirectly through item counts
  • Randomised response techniques use probability models for anonymity
  • Endorsement experiments assess support without direct attribution
  • Conjoint analysis measures preferences through trade-off scenarios
  • Sometimes requires careful probability weighting in analysis

Social desirability bias

Social desirability bias

  • Social desirability bias occurs when respondents provide answers they believe are more socially acceptable
  • This can lead to underreporting of sensitive behaviours or attitudes
  • Common examples include:
    • Political beliefs
    • Criminal behaviour
    • Discrimination
    • Substance use
    • Sexual behaviour
  • Self-censorship can also occur when respondents fear judgement
  • Social desirability bias can occur even in anonymous surveys, so it is important to use techniques that reduce bias
  • Researchers have developed various methods to mitigate this bias
  • Today we will see four of them:
    • List experiments
    • Randomised response techniques
    • Endorsement experiments
    • Conjoint analysis
  • All of them are well-established methods in survey research and have R packages available

List experiments 📋

What is a list experiment?

  • The logic of list experiments is simple
  • Respondents are presented with a list of items and asked how many they agree with
    • Just how many items, not which ones
  • The list includes a sensitive item (e.g., “I have committed a crime”) and several non-sensitive items
  • The sensitive item is randomly assigned to a subset of respondents
  • The key is to compare the average number of items agreed with between the treatment group (who sees the sensitive item) and the control group (who does not)
  • The difference in means provides an estimate of the prevalence of the sensitive item
  • This method is also known as the item count technique

Example

Measuring prejudice

Now I’m going to read you three things that sometimes make people angry or upset. After I read all three, just tell me HOW MANY of them upset you. (I don’t want to know which ones, just how many.)

  1. the federal government increasing the tax on gasoline
  2. professional athletes getting million-dollar-plus salaries
  3. large corporations polluting the environment

How many, if any, of these things upset you?

Example of a list experiment

Measuring prejudice

Now I’m going to read you four things that sometimes make people angry or upset. After I read all four, just tell me HOW MANY of them upset you. (I don’t want to know which ones, just how many.)

  1. the federal government increasing the tax on gasoline
  2. professional athletes getting million-dollar-plus salaries
  3. large corporations polluting the environment
  4. a Muslim family moving next door to you

How many, if any, of these things upset you?

Some notation

  • Sample of respondents \(N\), where \(T_i = 1\) if respondent \(i\) is in the treatment group and \(T_i = 0\) if in the control group
  • \(J\) is the number of items in the control list, \(J + 1\) is the number of items in the treatment list
  • \(Z_{ij}(t)\) a binary variable denoting respondent \(i\)’s preference for the \(j\)th control item for \(j = 1, \dots , J\) under the treatment status \(t = 0, 1\)
  • \(Y_{i}(0) = \sum_{j=1}^{J} Z_{ij}(0)\) is the potential answer \(i\) would give if asked about the control list
  • \(Y_{i}(1) = \sum_{j=1}^{J+1} Z_{ij}(1)\) is the number of items in the treatment list that respondent \(i\) agrees with
  • The observed response is \(Y_i = Y_i(T_i)\), where \(Y_i(0)\) is in the range of \(\{0,1, \dots, J\}\) and \(Y_i(1)\) is in the range of \(\{0,1, \dots J + 1\}\)
  • Now let’s discuss the assumptions…

Assumptions

No design effects

  • First, we need to assume that the addition of the sensitive item does not change the sum of affirmative answers to the control items
  • It is not necessary that respondents answer the control items truthfully, but the average number of affirmative answers must be the same in both groups
  • This is the no design effects assumption
  • Formally, for each respondent \(i = 1, \dots, N\), we assume:

\[ \sum_{j=1}^{J} Z_{ij}(0) = \sum_{j=1}^{J} Z_{ij}(1) \text{ or equivalently } Y_i(0) = Y_i(1) + Z_{i,J+1}(1). \]

No liars

  • Second, we need to assume that the sensitive item is not a lie
  • That is, all respondents give truthful answers for the sensitive item
  • This is a strong assumption, as you can imagine
  • Formally, we assume that:

\[ Z_{i,J+1}(1) = Z^*_{i,J+1} \]

where \(Z^*_{i,J+1}\) represents a truthful answer to the sensitive item. The treatment effect is

\[\hat{\tau} = \frac{1}{N_1} \sum_{i=1}^{N} T_i Y_i - \frac{1}{N_0} \sum_{i=1}^{N} (1 - T_i) Y_i,\]

where \(N_1 = \sum_{i=1}^{N} T_i\) is the size of the treatment group and \(N_0 = N - N_1\) is the size of the control group

More about list experiments

Notes about the design

  • List experiements have several advantages, as they are easy to implement and clear to respondents
  • But they have some issues as well:
    • Limited power to detect small effects
    • Floor effects: If many respondents disagree with all or most items, it is hard to estimate the prevalence of the sensitive item
    • Ceiling effects: If someone agrees with all items, we know for sure they agree with the sensitive item
    • Sample homogeneity: If the sample is too homogeneous, it may be hard to detect differences
  • But there are some solutions for these problems
  • Use items that contradict each other to reduce ceiling and floor effects
    • Example: ask about pro-gun and pro-choice items
  • A weakly informative Bayesian prior can also be used to reduce ceiling and floor effects,
  • Pre-treatment covariates can be used to improve power and precision
  • And there are more ways to improve the design too…